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Abstract. This paper reports on a research project in a university English as 
Foreign Language (EFL) program in Japan which explored ways to sustain active 
participation in e-leaming tasks. The tasks were intended to improve students’ 
scores on the Test of English for International Communication (TOEIC), a test used 
by businesses to make hiring decisions. The research adopted a Flipped Learning 
(FL) approach to Blended Learning (BL). A web-based courseware, ATR CALL 
BRIX (http://www.atr-lt.jp/products/brix/index.html), which featured e-learning 
materials for the TOEIC Test, was used. The students used mobile devices to access 
the courseware before class in order to prepare for in-class teacher- student analysis 
of their performance on the learning tasks. The teaching methodology integrated 
the online and in-class tasks in a single learning environment by means of an 
e-mentoring system used in conjunction with an in-class student self-evaluation 
task. The findings of pre- and post-TOEIC testing showed a significant degree of 
TOEIC score improvement in the experimental group. Post-course evaluations 
revealed that the combination of e-mentoring and the in-class self-evaluation system 
had encouraged sustained engagement in outside-of-class learning activities. 
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1. Introduction: flipped learning for foreign language education 

BL is a combination of conventional classroom face-to-face delivery of lesson 
instruction with online-mediated instruction (Osguthorpe & Graham, 2003). FL is 
a BL teaching methodology which uses the two BL channels of communication to 
reverse the conventional patterns of classroom teaching and learning. FL rests on 
an assumption that students will use learning task materials before class, so that 
classroom time can be devoted to interaction between students and their teacher 
that will support the outside-of-class study through mutual problem-solving tasks 
and analytical examinations of learning materials. This study examined a FL 
approach to generating, and then sustaining, student motivation that would result 
in effective long-term use of on-line EFL learning materials. 

Until recently, it has been expected that university students would mainly use 
laptop and desktop computers to access online learning materials at school and 
elsewhere. However, the use of a variety of mobile devices with online access, 
such as mobile phones and tablet computers, is now the preferred means of student- 
to-student online communication in some university communities. For example, 
94.9% of students who will graduate in Japan in 2016 have smartphones compared 
with 16.4% of the 2012 graduates. More than 85% of these students use one Social 
Network System, LINE, not only to communicate with their friends but also as a 
source of information. In sharp contrast, only 11.2% of the 2016 graduating class 
use desk top computers as their main means of information acquisition (Mynavi, 
2015). Thus, this research project explored a mobile assisted language learning 
medium of BL methodology. 

2. Course design: the EFL flipped learning course 

Figure 1. Procedure of activities in one session 
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In order to help students improve their TOEIC scores, an original two-semester 
EFL FL course was implemented from April 2014 to January 2015 at a university 
in Japan. Each learning unit had three phases: 1) pre-class online completion of 
TOEIC course materials; 2) individualized problem-solving instruction in class; 
and 3) post-class online completion of self-assessment and reflection learning tasks 
(see Figure 1). 

2.1. ATR CALL BRIX 

Outside of class, a www-based courseware, ATR CALL BRIX (http://www.atr- 
lt.jp/products/brix/index.html), which included a Learning Management System 
(LMS), was used. The LMS provided a variety of learning materials designed to 
prepare students for the TOEIC Test. Seven different functions were featured on 
the LMS: 1) study logs, 2) feedback on the achievement rates of student-set goals, 
3) records of the frequency of the use of the materials, 4) a record of time spent on 
learning, 5) a continuous update of the average score on the TOEIC learning tasks, 
6) an evaluation of students’ weak points and advice for further learning, and 7) 
students’ rankings in comparison with other students in the course (Ishikawa et al., 
2014). 

2.2. A student self-evaluation system 

A student self-evaluation system that was intended to contribute to the development 
of students’ self-regulated learning attitudes, skills and behavior, and thus, sustain 
student use of the learning materials, was integrated in the courseware. The system 
combined e-mentoring in the LMS outside of class, and weekly in-class self- 
evaluations as part of the course routine. At the beginning of the first semester, 
students were placed in three groups - high, mid and low - according to their 
TOEIC scores. An e-mentoring team of one teacher and a teaching assistant sent 
different need-based messages of advice and encouragement to the students every 
week by means of a social networking system called LINE (http://line.me/en/). The 
messages were varied according to how successfully the students had completed 
the learning materials (Ishikawa et al., 2015). 

3. Validation of the study 

3.1. Participants 

551 first-year students who enrolled in a TOEIC course participated in this study. 
348 students were included in an experimental group and 203 students in a control 
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group. The students in the experimental group were placed in three groups - high, 
mid and low (mean ± 0.5SD ) - according to their scores of the TOEIC Test which 
they took at the beginning of the course. 

3.2. Methods 

For the experimental group, the course design described in section 2, above, was 
implemented. In the control group, students followed the same course design but 
they used neither the web-based courseware, ATR CALL BRIX, nor the self- 
evaluation system. All their learning materials were paper-based. 

Pre- and post-TOEIC Tests were conducted before the course started and at the end 
of the second semester in January 2015. The experimental-group students took the 
TOEIC Test again at the end of the first semester in July 2014, in addition to the 
pre- and post-TOEIC Test. The students also completed course evaluations at the 
end of the second semester. 

4. Results and discussion 

4.1. TOEIC testing 

The increase of TOEIC scores between the mid and low groups in the experimental 
group and the control group was compared. The high group in the experimental 
group was excluded because the average scores for the pre-test in both 
experimental and control groups should be comparable. The increase of the scores 
in the experimental group was 15 1 .38 and that in the control group was 54.04. The 
scores of the post-test in the two groups was compared and there was a significant 
difference (p<.001; d=. 56) as is shown in Table 1. 


Table 1. Results of pre- and post-TOEIC testing 



Experimental 



Control 


t 

d 


Mean 

SD 

n 

Mean 

SD 

n 

Pre-TOEIC (April) 

345.97 

66.82 

243 

361.03 

89.14 

203 

2.00* 

0.14 

Post-TOEIC (January) 

497.35 

107.77 

243 

415.07 

107.20 

203 

8.05 *** 

0.56 


*p<. 05, ***/X.001 


Furthermore, students who took the TOEIC Test in April and July in 2014, and 
January in 2015 in each of the three groups of the experimental group improved 
their TOEIC scores by approximately 150 points from April in 2014 to January in 
2015 as is shown in Table 2. 
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Table 2. Results of TOEIC testing of the three groups in the experimental group 



April 


July 


January 


Mean 

SD 


Mean SD 


Mean 

SD 

High 

511.95 

61.21 


603.38 81.75 


652.38 

86.43 

Intermediate 

393.89 

28.71 


507.17 71.91 


543.44 

96.14 

Low 

286.12 

43.65 


409.58 79.17 


443.50 

96.35 


105 

122 

107 


4.2. Post course student evaluation 


The course evaluation for the experimental group conducted at the end of the second 
semester consisted of 20 questions (see Table 3). Four open-ended questions asked 
about the students’ feelings about the course and the in-class activities, the problems 
that they may have found in using the LMS learning materials, and the messages 
sent by the e-mentor team. A 4-point Likert scale was used for the responses in 
order to adequately allow for the expression of a range of participants’ feelings 
about the course. The ratings of 2 and 1 respectively corresponded to disagree, 
and strongly disagree; and the ratings of 3 and 4 respectively corresponded to 
agree, and strongly agree. The rate of reliability of the twenty 4-point Likert scale 
questions was high (a=.91). 


Table 3. Results of the questions in the participants’ course evaluation 


Item 

Mean 

SD 

1 The in-class activites helped improve my total TOEIC score. 

3.26 

0.79 

2 The in-class activites helped improve my TOEIC listening score. 

3.24 

0.81 

3 The in-class activites helped improve my TOEIC reading score. 

3.17 

0.84 

4 The in-class activites helped improve my vocabulary. 

2.90 

0.84 

5 The in-class activites helped improve my English listening skills. 

3.19 

0.82 

6 The in-class activites helped improve my English reading skills. 

3.00 

0.85 

7 In the in-class activities, I was able to solve problems that I found in the outside-of-class activities. 

2.76 

0.84 

8 I was interested in the in-class activities. 

2.82 

0.89 

9 I want to continue the in-class activities. 

2.80 

0.93 

10 I used the learning materials in the LMS of the ATR CALL BRIX. 

3.38 

0.82 

1 1 The learning materials in the IMS of the ATR CALL BRIX helped improve my total TOEIC score. 

2.91 

0.90 

12 The learning materials in the IMS of the ATR CALL BRIX helped improve my TOEIC listening score. 

3.05 

0.86 

13 The learning materials in the LMS of the ATR CALL BRIX helped improve my TOEIC reading s core. 

2.77 

0.93 

14 The learning materials in the IMS of the ATR CALL BRIX helped improve my vocabulary. 

2.88 

0.91 

15 The learning materials in the IMS ofthe ATR CALL BRIX helped improve my English listening skills. 

3.00 

0.89 

16 The learning materials in the IMS of the ATR CALL BRIX helped improve my English reading skills . 

2.81 

0.94 

17 I was interested in using the learning materials in the LMS of the ATR CALL BRIX. 

2.50 

0.98 

18 I want to continue using the learning materials in the LMS of the ATR CALL BRIX. 

2.49 

1.05 

19 I often found some problems using the learning materials in the IMS ofthe ATR CALL BRIX. 

2.51 

1.03 

20 The flipped learning lessons suited my own learning style. 

2.67 

0.87 

n =331 


As for the in-class activities, the participants were convinced that the activities 
were useful for strengthening their TOEIC listening and reading skills. Typical 
comments were similar to the following student’s response: “I enjoyed the in- 
class activities. I was able to learn TOEIC test- taking strategies to improve my 
TOEIC scores”. However, some participants did not feel that they were able to 
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solve problems that they had faced while using the LMS learning materials: “I 
don’t know why but I often had problems in using the learning materials”. There 
were other negative reactions to the use of the LMS: looking at the display screen 
hurt their eyes; and they found problems in using the learning materials at home. 

However, the majority of the participants felt that the learning materials in the 
LMS helped improve their TOEIC scores. Some of the participants believed that 
the FL approach in the study suited their own learning styles: “The encouragement 
from the teacher let me continue the outside-of-class activities, I really think so”. 

5. Conclusion 

The results of the pre- and post-TOEIC testing showed that the EFL FL course design 
for the experimental group was more effective in helping students improve their 
TOEIC scores than the course design for the control group, and that the students at 
all TOEIC-score levels in the experimental group improved their TOEIC scores by 
an average of approximately 150 points. Moreover, the study revealed that the self- 
evaluation system encouraged students to sustain engagement in outside-of-class 
e-learning learning activities. Further research should be conducted to investigate 
the most productive uses of mobile devices for e-learning. 
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